Retrieving product information from embedded sensors via mobile device video analysis

ABSTRACT

A system for accessing product information from embedded sensors via mobile device video analysis is disclosed. The system includes a processing device configured for analyzing a video stream of the surrounding environment captured on a mobile device. At least one product that has one or more embedded sensors associated therewith is identified within the surrounding environment. The system retrieves information associated with the one or more products from the one or more embedded sensors. Information retrieved, or information derived from the information retrieved, from the embedded sensors is presented to the user via a display of the video stream on the mobile device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/450,213, filed Mar. 8, 2011, entitled “Real-Time Video ImageAnalysis Applications for Commerce Activity,” and U.S. ProvisionalPatent Application Ser. No. 61/478,411, filed Apr. 22, 2011, entitled“Retrieving Product Information From Embedded Sensors Via Mobile DeviceVideo Analysis,” the entirety of each of which is incorporated herein byreference.

BACKGROUND

Modern handheld mobile devices, such as smart phones or the like,combine multiple technologies to provide the user with a vast array ofcapabilities. For example, many smart phones are equipped withsignificant processing power, sophisticated multi-tasking operatingsystems, and high-bandwidth Internet connection capabilities. Moreover,such devices often have additional hardware and/or software that providefeatures and functionality that are becoming increasingly more commonand standardized. Such features and functionality include, but are notlimited to, location-determining functionality, such as GlobalPositioning System (GPS) functionality; hardware and/or software forsensing operational conditions of the devices or even the ambientconditions in which the devices are located, such as accelerometers andhigh-resolution video cameras.

As the hardware capabilities of such mobile devices have increased, sotoo have the software applications that control and operate thehardware. One such example of innovative software is a category known asaugmented reality (AR), or more generally referred to as mediatedreality. One such example of an AR application platform is Layar®,available from Layar®, Amsterdam, the Netherlands.

The Layar® application platform analyzes location data and compassdirection data in combination with information related to the objects,locations or other elements shown in the video stream to createbrowse-able “hot-spots” or “tags” that are superimposed on the mobiledevice display, resulting in an experience described as “realitybrowsing”.

However, the Layar® application platform strictly utilizes GPS andcompass data in order to create the “reality browsing.” The Layar®application platform is incapable of actually identifying objects withina video stream. Furthermore, the Layar® application platform is unableto retrieve and present data from, for example, a sensor that may beembedded in a consumer product.

Therefore, a need exists to implement video analysis and present it tothe user as AR or the like, to assist the user of mobile devices withaccessing product information from embedded sensors.

SUMMARY OF INVENTION

The following presents a summary of certain embodiments of the presentinvention. This summary is not intended to be a comprehensive overviewof all contemplated embodiments, and is not intended to identify all keyor critical elements of all embodiments nor delineate the scope of anyor all embodiments. Its sole purpose is to present certain concepts andelements of one or more embodiments in a summary form as a prelude tothe more detailed description that follows.

Methods, apparatuses, systems and computer program products aredescribed herein that provide for using video analysis to process videostreams and, more specifically, to access product information fromembedded sensors and present information associated with the product tothe user, which information may be presented via AR or the like. Theinformation presented may be information retrieved or obtained from theembedded sensors or it may be information compiled using the informationretrieved or obtained from the embedded sensors. The methods,apparatuses, systems, and computer program products may additionallyprovide for alerting the user to important or critical informationassociated with one or more products. In one embodiment, the user ispresented with information pertaining to, and/or alerted to, importantor critical product information, such as the depletion or expiration ofa product that requires replacement or will require replacement in thenear future or a requirement to obtain service for the product. In oneembodiment, the embedded sensors may provide information necessary orbeneficial in order to present desired information to the user. Inanother embodiment, through the use of vision object recognition,objects, logos, artwork, products, locations and other features of aproduct (referred to herein as attributes) that can be recognized fromthe surrounding environment represented in the video stream and that canbe identified and matched to stored data or data that can be accessed inreal-time (collectively, “preexisting data”) in order to identify aproduct or information pertaining to the product or the user. In oneembodiment, the preexisting data may comprise customer financialbehavior history, customer credit score, customer purchasepower/transaction history, wish lists, shopping lists, to do/task list,vendor information (contact information, account information, paymentrequirements and acceptable forms of payment, etc.), informationregarding products and services (e.g., costs, availability, etc.),and/or other financial information concerning the customer. In thisregard, embodiments of the present invention leverage institutional orproprietary financial data that is uniquely specific to a particularfinancial institution to provide information to users in connection withthe video stream analysis.

To accomplish the foregoing and related objectives, the embodiments ofthe present invention comprise the function and features hereinafterdescribed. The following description and the referenced figures setforth a detailed description of the present invention, including certainillustrative examples of the one or more embodiments. The functions andfeatures described herein are indicative, however, of but a few of thevarious ways in which the principles of the present invention may beimplemented and used and, thus, this description is intended to includeall such embodiments and their equivalents.

In one embodiment of the present invention, a method is disclosed foraccessing product information from embedded sensors. The method includesanalyzing, via a processing device, a video stream representing thesurrounding environment captured on a mobile device. The method furtherincludes identifying one or more products within the video stream thathas one or more embedded sensors associated therewith. Additionally, themethod includes retrieving information associated with the one or moreproducts from the one or more embedded sensors. Finally, the methodincludes presenting, via a display of the video stream on the mobiledevice, the information retrieved or obtained, or information derivedfrom the information retrieved or obtained, from the one or moreembedded sensors.

In one embodiment of the method, the mobile device is a mobile phone.

In one embodiment of the method, the mobile device includes a cameracapable of image and/or video capture.

In one embodiment of the method, the one or more embedded sensors emit aradio frequency signal. In yet other embodiments of the method, the oneor more embedded tags emit an other wave frequencies, such as infraredor X-ray.

Additionally, in one embodiment of the method, the mobile device isoperated in the “always on” mode to continuously analyze the surroundingenvironment.

In one embodiment of the method, the information presented is displayedas a virtual object overlaying a real-time video stream of thesurrounding environment. In one embodiment, the virtual object isinteractive.

In still another embodiment, the method further includes alerting theuser to important or critical information associated with one or moreproducts. Alerting the user may be accomplished by producing an audiblesound, vibration, generating a Short Message Service or text message,generating an email, or combinations thereof. The important or criticalinformation may include information regarding a product default or animpending product default. In one embodiment, the product default is thedepletion or expiration of the product requiring replacement. In anotherembodiment, the product default is a mechanical default requiringmaintenance or repair services.

In one embodiment of the method, the one or more products is a foodproduct. In another embodiment, the one or more products is a householditem such as a household appliance, vehicle, monitoring systems, sensorsin the handsets broadcasting the user's location for other friends toobserve and/or discover.

In one embodiment, the method further includes presenting the user withan option to purchase the one or more products via the mobile device. Inanother embodiment, the method further includes transmitting paymentinformation via a wireless connection between the mobile device and acontactless payment terminal to purchase a product.

In one embodiment, the method further includes transmitting financialdata and/or other user-specific data to a financial institution.

In another embodiment of the present invention, a system for accessingproduct information from embedded sensors is disclosed. The systemincludes a processing device configured for analyzing a video streamrepresenting the surrounding environment captured on a mobile device.The processing device is further configured for identifying one or moreproducts within the video stream that has one or more embedded sensorsassociated therewith. Additionally, the processing device includesretrieving or obtaining information associated with the one or moreproducts from the one or more embedded sensors. Finally, the processingdevice is further configured for presenting, via a display of the videostream on the mobile device, the information retrieved or obtained fromthe one or more embedded sensors.

In one embodiment of the system, the mobile device is a mobile phone.

In one embodiment of the system, the mobile device includes a cameracapable of image and/or video capture.

In one embodiment of the system, the one or more embedded sensors emit aradio frequency signal.

Additionally, in one embodiment of the system, the mobile device isoperated in the “always on” mode to continuously analyze the surroundingenvironment.

In one embodiment of the system, the information presented is displayedas a virtual object overlaying a real-time video stream of thesurrounding environment. In some such embodiments, the virtual object isinteractive.

In still another embodiment, the system further includes alerting theuser to important or critical information associated with one or moreproducts. Alerting the user may be accomplished by producing an audiblesound, vibration, generating a Short Message Service or text message,generating an email, phone message, or combinations thereof. Theimportant or critical information may include information regarding aproduct default or an impending product default. In one embodiment ofthe system, the product default is the depletion or expiration of theproduct requiring replacement. In another embodiment of the system, theproduct default is a mechanical default requiring maintenance.

In one embodiment of the system, the one or more products is a foodproduct. In another embodiment, the one or more products is a householditem such as a household appliance.

Furthermore, in one embodiment, the system further includes presentingthe user with an option to purchase the one or more products via themobile device. In still a further embodiment, the system furtherincludes transmitting payment information via a wireless connectionbetween the mobile device and a contactless payment terminal to purchasea product.

In another embodiment of the present invention, a computer programproduct for accessing product information from embedded sensors includesa non-transitory computer readable medium including one or morecomputer-readable code portions configured for analyzing a video streamof the surrounding environment captured on a mobile device. The one ormore computer-readable code portions are further configured foridentifying one or more products within the video stream that has one ormore embedded sensors. Additionally, the one or more computer-readablecode portions are further configured to retrieve or obtain informationassociated with the one or more products from the one or more embeddedsensors. Finally, the one or more computer-readable code portions arefurther configured to present, via a display of the video stream on themobile device, the information retrieved, or information derived fromthe information retrieved, from the one or more embedded sensors.

In one embodiment of the computer program product, the mobile device isa mobile phone.

In one embodiment of the computer program product, the mobile deviceincludes a camera capable of image and/or video capture.

In one embodiment of the computer program product, the one or moreembedded sensors emit a radio frequency signal or other light signal.

Additionally, in one embodiment of the computer program product, themobile device is operated in the “always on” mode to continuouslyanalyze the surrounding environment.

In one embodiment of the computer program product, the informationpresented is displayed as a virtual object overlaying a real-time videostream of the surrounding environment. In one embodiment, the virtualobject is interactive.

In still another embodiment of computer program product, one or morecomputer-readable code portions are further configured for alerting theuser to important or critical information associated with one or moreproducts. Alerting the user may be accomplished by producing an audiblesound, vibration, generating a Short Message Service or text message,generating an email, or combinations thereof. The important or criticalinformation may include information regarding a product default or animpending product default. In one embodiment of the computer programproduct, the product default is the depletion or expiration of theproduct requiring replacement.

In another embodiment of the system, the product default is a mechanicaldefault requiring maintenance or repair services.

In one embodiment of the computer program product, the one or moreproducts is a food product. In another embodiment, the one or moreproducts is a household item such as a household appliance.

In one embodiment of the computer program product, the one or morecomputer-readable code portions if further configured to present theuser with an option to purchase the one or more products via the mobiledevice. In still a further embodiment, the system further includestransmitting payment information via a wireless connection between themobile device and a contactless payment terminal to purchase a product.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described embodiments of the invention in general terms,reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 2 is a block diagram illustrating an AR environment, in accordancewith an embodiment of the invention;

FIG. 3 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 4 is a flow diagram illustrating a method for accessing productinformation from embedded sensors, in accordance with an embodiment ofthe invention;

FIG. 5 is a mixed block and flow diagram illustrating a system foraccessing product information from embedded sensors, in accordance withan embodiment of the invention;

FIG. 6 is a block diagram illustrating a contactless paymentenvironment, in accordance with an embodiment of the invention;

FIG. 7 is a block diagram illustrating the contactless payment terminalof FIG. 6, in accordance with an embodiment of the invention; and

FIG. 8 illustrates a mobile wallet, in accordance with an embodiment ofthe invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all, embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more embodiments. It may be evident;however, that such embodiment(s) may be practiced without these specificdetails. Like numbers refer to like elements throughout.

Various embodiments or features will be presented in terms of systemsthat may include a number of devices, components, modules, and the like.It is to be understood and appreciated that the various systems mayinclude additional devices, components, modules, etc. and/or may notinclude all of the devices, components, modules etc. discussed inconnection with the figures. A combination of these approaches may alsobe used.

The steps and/or actions of a method or algorithm described inconnection with the embodiments disclosed herein are directed to sensorsassociated with the product. A sensor may sense or read a productstatus. In this way, a sensor may be any type of sensory device,including but not limited to a tag, RFID-chip, radio signal emitter, ormarker embedded in a customer product that may provide an indication asto the status of a product. The status of a product may include theamount of the product used by the customer. For example, the status of amilk jug may be an indication as to how much milk is remaining in acontainer of milk stored in the customer's refrigerator. In someembodiments, the terms sensor or sensors are intended to refer to asensor working in conjunction with one or more other devices. Forexample, in some embodiments, one or more tags are disposed in, on orproximate a product or other object and a sensor remote from tag readsthe tag.

The steps and/or actions of a method or algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware, in one or more software modules (also referred toherein as computer-readable code portions) executed by a processor orprocessing device and configured for performing certain functions, or ina combination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a harddisk, a removable disk, a CD-ROM, or any other form of non-transitorystorage medium known in the art. An exemplary storage medium may becoupled to the processing device, such that the processing device canread information from, and write information to, the storage medium. Inthe alternative, the storage medium may be integral to the processingdevice. Further, in some embodiments, the processing device and thestorage medium may reside in an Application Specific Integrated Circuit(ASIC). In the alternative, the processing device and the storage mediummay reside as discrete components in a computing device. Additionally,in some embodiments, the events and/or actions of a method or algorithmmay reside as one or any combination or set of codes or code portionsand/or instructions on a machine-readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

In one or more embodiments, the functions described may be implementedin hardware, software, firmware, or any combination thereof. Ifimplemented in software, the functions may be stored or transmitted asone or more instructions, code, or code portions on a computer-readablemedium. Computer-readable media includes both non-transitory computerstorage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another. Astorage medium may be any available media that can be accessed by acomputer. By way of example, and not limitation, such computer-readablemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother medium that can be used to carry or store desired program code inthe form of instructions or data structures, and that can be accessed bya computer. Also, any connection may be termed a computer-readablemedium. For example, if software is transmitted from a website, server,or other remote source using a coaxial cable, fiber optic cable, twistedpair, digital subscriber line (DSL), or wireless technologies such asinfrared, radio, and microwave, then the coaxial cable, fiber opticcable, twisted pair, DSL, or wireless technologies such as infrared,radio, and microwave are included in the definition of medium. “Disk”and “disc”, as used herein, include compact disc (CD), laser disc,optical disc, digital versatile disc (DVD), floppy disk and blu-ray discwhere disks usually reproduce data magnetically, while discs usuallyreproduce data optically with lasers. Combinations of the above shouldalso be included within the scope of computer-readable media.

In accordance with embodiments of the invention, the term “financialinstitution” refers to any organization in the business of moving,investing, or lending money, dealing in financial instruments, orproviding financial services. This includes commercial banks, thrifts,federal and state savings banks, savings and loan associations, creditunions, investment companies, merchants, insurance companies and thelike.

Thus, methods, systems, computer programs and the like are hereindisclosed that provide for using video analysis to process video streamsand, more specifically, to access product information from embeddedsensors and present information associated with the product to the user,which information may be presented via AR or the like. The methods,apparatus, systems, and computer program products may additionallyprovide for alerting the user to important or critical informationassociated with one or more products. In one embodiment, the user ispresented with information pertaining to, and/or alerted to, importantor critical product information, such as the depletion or expiration ofa product that requires replacement or will require replacement in thenear future. In one embodiment, the embedded sensors may provide all ofthe necessary information in order to present desired information to theuser. However, in another embodiment, through the use of vision objectrecognition, objects, logos, artwork, products, locations and otherfeatures of a product (referred to herein as attributes) that can berecognized from the surrounding environment represented in the videostream can be identified and matched to stored data or data that can beaccessed in real-time (collectively, “preexisting data”). In oneembodiment, the preexisting data is specific to one or more financialinstitutions. For example, the preexisting data may comprise customerfinancial behavior history, customer credit score, customer purchasepower/transaction history, vendor information (contact information,account information, payment requirements and acceptable forms ofpayment, etc.), information regarding products and services (e.g.,costs, availability, etc.), and/or other financial informationconcerning the customer. In this regard, embodiments of the presentinvention leverage institutional or proprietary financial data that isuniquely specific to a particular financial institution to provideinformation to users in connection with the video stream analysis.

While embodiments discussed herein are generally described with respectto “video streams” or “real-time video” it will be appreciated that thevideo stream may be captured and stored for later viewing and analysis.Indeed, in some embodiments video is recorded and stored on a mobiledevice and portions or the entirety of the video may be analyzed at alater time. The later analysis may be conducted on the mobile device orloaded onto a different device for analysis. The portions of the videothat may be stored and analyzed may range from a single frame of video(e.g., a screenshot) to the entirety of the video. Additionally, ratherthan video, the user may opt to take a still picture of the environmentto be analyzed immediately or at a later time. Thus, embodiments inwhich real-time video, recorded video or still pictures are analyzed arecontemplated herein.

FIG. 1 illustrates an embodiment of a mobile device 100 that may beconfigured to execute object recognition and Augmented Reality (AR)functionality, in accordance with specific embodiments of the presentinvention. A “mobile device” 100 may be any mobile communication device,such as a cellular telecommunications device (i.e., a cell phone ormobile phone), personal digital assistant (PDA), a mobile Internetaccessing device, or other mobile device including, but not limited toportable digital assistants (PDAs), pagers, mobile televisions, gamingdevices, laptop computers, cameras, video recorders, audio/video player,radio, GPS devices, any combination of the aforementioned, or the like.

The mobile device 100 may generally include a processor 110 communicablycoupled to such devices as a memory 120, user output devices 136, userinput devices 140, a network interface 160, a power source 115, a clockor other timer 150, an image capture device 170, a positioning systemdevice 175 (e.g., a Global Positioning System (GPS) device), one or moreintegrated circuits 180, etc.

In some embodiments, the mobile device and/or the server access one ormore databases or data stores (not shown in FIG. 1) to search for and/orretrieve information related to the object and/or marker. In someembodiments, the mobile device and/or the server access one or more datastores local to the mobile device and/or server and in otherembodiments, the mobile device and/or server access data stores remoteto the mobile device and/or server. In some embodiments, the mobiledevice and/or server access both a memory and/or data store local to themobile device and/or server as well as a data store remote from themobile device and/or server.

The processor 110, and other processors described herein, may generallyinclude circuitry for implementing communication and/or logic functionsof the mobile device 100. For example, the processor 110 may include adigital signal processor device, a microprocessor device, and variousanalog to digital converters, digital to analog converters, and/or othersupport circuits. Control and signal processing functions of the mobiledevice 100 may be allocated between these devices according to theirrespective capabilities. The processor 110 thus may also include thefunctionality to encode and interleave messages and data prior tomodulation and transmission. The processor 110 may additionally includean internal data modem. Further, the processor 110 may includefunctionality to operate one or more software programs or applications,which may be stored in the memory 120. For example, the processor 110may be capable of operating a connectivity program, such as a webbrowser application 122. The web browser application 122 may then allowthe mobile device 100 to transmit and receive web content, such as, forexample, location-based content and/or other web page content, accordingto a Wireless Application Protocol (WAP), Hypertext Transfer Protocol(HTTP), and/or the like.

The processor 110 may also be capable of operating applications, such asan object recognition application 125. The object recognitionapplication 14 may be downloaded from a server and stored in the memory120 of the mobile device 100. Alternatively, the object recognitionapplication 125 may be pre-installed and stored in a memory in theintegrated circuit 180. In such an embodiment, the user may not need todownload the object recognition application 125 from a server. In someembodiments, the processor 110 may also be capable of operating one ormore applications, such as one or more applications functioning as anartificial intelligence (“AI”) engine. The processor 110 may recognizeobjects that it has identified in prior uses by way of the AI engine. Inthis way, the processor 110 may recognize specific objects and/orclasses of objects, and store information related to the recognizedobjects in one or more memories and/or databases discussed herein. Oncethe AI engine has thereby “learned” of an object and/or class ofobjects, the AI engine may run concurrently with and/or collaborate withother modules or applications described herein to perform the varioussteps of the methods discussed. For example, in some embodiments, the AIengine recognizes an object that has been recognized before and storedby the AI engine. The AI engine may then communicate to anotherapplication or module of the mobile device and/or server, an indicationthat the object may be the same object previously recognized. In thisregard, the AI engine may provide a baseline or starting point fromwhich to determine the nature of the object. In other embodiments, theAI engine's recognition of an object is accepted as the finalrecognition of the object.

The integrated circuit 180 may include the necessary circuitry toprovide the object recognition functionality to the mobile device 100.Generally, the integrated circuit 180 will include data storage 171which may include data associated with the objects within a video streamthat the object recognition application 125 identifies as having acertain marker(s) (discussed in relation to FIG. 2). The integratedcircuit 180 and/or data storage 171 may be an integrated circuit, amicroprocessor, a system-on-a-integrated circuit, a microcontroller, orthe like. As discussed above, in one embodiment, the integrated circuit180 may provide the functionality to the mobile device 100.

Of note, while FIG. 1 illustrates the integrated circuit 180 as aseparate and distinct element within the mobile device 100, it will beapparent to those skilled in the art that the object recognitionfunctionality of integrated circuit 180 may be incorporated within otherelements in the mobile device 100. For instance, the functionality ofthe integrated circuit 180 may be incorporated within the mobile devicememory 120 and/or processor 110. In a particular embodiment, thefunctionality of the integrated circuit 180 is incorporated in anelement within the mobile device 100 that provides object recognitioncapabilities to the mobile device 100. Still further, the integratedcircuit 180 functionality may be included in a removable storage devicesuch as an SD card or the like.

The processor 110 may be configured to use the network interface 160 tocommunicate with one or more other devices on a network. In this regard,the network interface 160 may include an antenna 176 operatively coupledto a transmitter 174 and a receiver 172 (together a “transceiver”). Theprocessor 110 may be configured to provide signals to and receivesignals from the transmitter 174 and receiver 172, respectively. Thesignals may include signaling information in accordance with the airinterface standard of the applicable cellular system of the wirelesstelephone network that may be part of the network. In this regard, themobile device 100 may be configured to operate with one or more airinterface standards, communication protocols, modulation types, andaccess types. By way of illustration, the mobile device 100 may beconfigured to operate in accordance with any of a number of first,second, third, and/or fourth-generation communication protocols and/orthe like. For example, the mobile device 100 may be configured tooperate in accordance with second-generation (2G) wireless communicationprotocols IS-136 (time division multiple access (TDMA)), GSM (globalsystem for mobile communication), and/or IS-95 (code division multipleaccess (CDMA)), or with third-generation (3G) wireless communicationprotocols, such as Universal Mobile Telecommunications System (UMTS),CDMA2000, wideband CDMA (WCDMA) and/or time division-synchronous CDMA(TD-SCDMA), with fourth-generation (4G) wireless communicationprotocols, and/or the like. The mobile device 100 may also be configuredto operate in accordance with non-cellular communication mechanisms,such as via a wireless local area network (WLAN) or othercommunication/data networks.

The network interface 160 may also include an object recognitioninterface 173 in order to allow a user to execute some or all of theabove-described processes with respect to the object recognitionapplication 125 and/or the integrated circuit 180. The objectrecognition interface 173 may have access to the hardware, e.g., thetransceiver, and software previously described with respect to thenetwork interface 160. Furthermore, the object recognition interface 173may have the ability to connect to and communicate with an external datastorage on a separate system within the network as a means ofrecognizing the object(s) in the video stream.

As described above, the mobile device 100 may have a user interface thatincludes user output devices 136 and/or user input devices 140. The useroutput devices 136 may include a display 130 (e.g., a liquid crystaldisplay (LCD) or the like) and a speaker 132 or other audio device,which are operatively coupled to the processor 110. The user inputdevices 140, which may allow the mobile device 100 to receive data froma user, may include any of a number of devices allowing the mobiledevice 100 to receive data from a user, such as a keypad, keyboard,touch-screen, touchpad, microphone, mouse, joystick, other pointerdevice, button, soft key, and/or other input device(s).

The mobile device 100 may further include a power source 115. Generally,the power source 115 is a device that supplies electrical energy to anelectrical load. In one embodiment, power source 115 may convert a formof energy such as solar energy, chemical energy, mechanical energy, etc.to electrical energy. Generally, the power source 115 in a mobile device100 may be a battery, such as a lithium battery, a nickel-metal hydridebattery, or the like, that is used for powering various circuits, e.g.,the transceiver circuit, and other devices that are used to operate themobile device 100. Alternatively, the power source 115 may be a poweradapter that can connect a power supply from a power outlet to themobile device 100. In such embodiments, a power adapter may beclassified as a power source “in” the mobile device.

The mobile device 100 may also include a memory 120 operatively coupledto the processor 110. As used herein, memory may include any computerreadable medium configured to store data, code, or other information.The memory 120 may include volatile memory, such as volatile RandomAccess Memory (RAM) including a cache area for the temporary storage ofdata. The memory 120 may also include non-volatile memory, which can beembedded and/or may be removable. The non-volatile memory mayadditionally or alternatively include an electrically erasableprogrammable read-only memory (EEPROM), flash memory or the like.

The memory 120 may store any of a number of applications or programswhich comprise computer-executable instructions/code executed by theprocessor 110 to implement the functions of the mobile device 100described herein. For example, the memory 120 may include suchapplications as an object recognition application 125, an augmentedreality (AR) presentation application 121 (described infra. in relationto FIG. 3), a web browser application 122, a Short Message Service (SMS)application 123, an electronic mail (i.e., email) application 124, etc.

Referring to FIG. 2, a block diagram illustrating an object recognitionexperience 200 in which a user 210 utilizes a mobile device 100 tocapture a video stream that includes an environment 250 is shown. Asdenoted earlier, the mobile device 100 may be any mobile communicationdevice. The mobile device 100 has the capability of capturing a videostream of the surrounding environment 250. The video capture may be byany means known in the art. In one particular embodiment, the mobiledevice 100 is a mobile telephone equipped with an image capture device170 capable of video capture.

The environment 250 contains a number of objects 220. Some of suchobjects 220 may include a marker 230 identifiable to an objectrecognition application that is either executed on the mobile device 100or within the wireless network. A marker 230 may be any type of markerthat is a distinguishing feature that can be interpreted by the objectrecognition application to identify specific objects 220. For instance,a marker 230 may be alpha-numeric characters, symbols, logos, shapes,ratio of size of one feature to another feature, a product identifyingcode such as a bar code, electromagnetic radiation such as radio waves(e.g., radio frequency identification (RFID)), architectural features,color, etc. In some embodiments, the marker 230 may be audio and themobile device 100 may be capable of utilizing audio recognition toidentify words or unique sounds broadcast. The marker 230 may be anysize, shape, etc. Indeed, in some embodiments, the marker 230 may bevery small relative to the object 220 such as the alpha-numericcharacters that identify the name or model of an object 220, whereas, inother embodiments, the marker 230 is the entire object 220 such as theunique shape, size, structure, etc.

In some embodiments, the marker 230 is not actually a physical markerlocated on or being broadcast by the object. For instance, the marker230 may be some type of identifiable feature that is an indication thatthe object 220 is nearby (e.g., a tag or placard near the object). Insome embodiments, the marker 230 for an object 220 may actually be themarker 230 for a different object 220. For example, the mobile device100 may recognize a particular building as being “Building A.” Datastored in the object recognition data storage may indicate that“Building B” is located directly to the east and next to “Building A.”Thus, markers 230 for an object 220 that are not located on or beingbroadcast by the object 220 are generally based on fixed facts about theobject 220 (e.g., “Building B” is next to “Building A”). However, it isnot a requirement that such a marker 230 be such a fixed fact. Themarker 230 may be anything that enables the mobile device 100 tointerpret to a desired confidence level what the object is. For anotherexample, the mobile device 100 may identify a particular person as afirst character from a popular show, and thereafter utilize theinformation that the first character is nearby features of othercharacters to interpret that a second character, a third character, etc.are nearby, whereas without the identification of the first character,the features of the second and third characters may not have been usedto identify the second and third characters. This example may also beapplied to objects outside of people.

The marker 230 may also be or include social network data, such as dataretrieved or communicated from the Internet, such as tweets, blog posts,Facebook posts, various types of messages and/or the like. In otherembodiments, the marker 230 is provided in addition to social networkdata as mentioned above. For example, the mobile device 100 may capturea video stream and/or one or more still shots including a largegathering of people. In this example, as above, one or more peopledressed as characters in costumes may be present at a specifiedlocation. The mobile device 100, object recognition application 121,and/or the AR presentation application 125 may identify several socialnetwork indicators, such as posts, blogs, tweets, messages, and/or thelike indicating the presence of one or more of the characters at thespecified location. In this way, the mobile device 100 and associatedapplications may communicate information regarding the social mediacommunications to the user and/or use the information regarding thesocial media communications in conjunction with other methods of objectrecognition. For example, the mobile device 100 object recognitionapplication 121, and/or the AR presentation application 125 performingrecognition of the characters at the specified location may confirm thatthe characters being identified are in fact the correct characters basedon the retrieved social media communications. This example may also beapplied objects outside of people.

In some embodiments, the mobile device 100 and/or server access one ormore other servers, social media networks, applications and/or the likein order to retrieve and/or search for information useful in performingan object recognition. In some embodiments, the mobile device 100 and/orserver accesses another application by way of an application programminginterface or API. In this regard, the mobile device 100 and/or servermay quickly search and/or retrieve information from the other programwithout requiring additional authentication steps or other gatewaysteps.

While FIG. 2 illustrates that the objects 220 with markers 230 onlyinclude a single marker 230, it will be appreciated that the object 220may have any number of markers 230 with each equally capable ofidentifying the object 220. Similarly, multiple markers 66 may beidentified by the mobile device 100 and associated applications suchthat the combination of the markers 230 may be utilized to identify theobject 220. For example, the mobile device 100 may utilize facialrecognition markers 230 to identify a person and/or utilize a separatemarker 230, such as the clothes the person is wearing to confirm theidentification to the desired confidence level that the person is infact the person the mobile device identified. For example, the facialrecognition may identify a person as a famous athlete, and thereafterutilize the uniform the person is wearing to confirm that it is in factthe famous athlete.

In some embodiments, a marker 230 may be the location of the object 220.In such embodiments, the mobile device 100 may utilize GlobalPositioning System (GPS) hardware and/or software or some other locationdetermining mechanism to determine the location of the user 210 and/orobject 220. As noted above, a location-based marker 230 could beutilized in conjunction with other non-location-based markers 230identifiable and recognized by the mobile device 100 to identify theobject 220. However, in some embodiments, a location-based marker may bethe only marker 230. For instance, in such embodiments, the mobiledevice 100 may utilize GPS software to determine the location of theuser 210 and a compass device or software to determine what directionthe mobile device 100 is facing in order to identify the object 220. Instill further embodiments, the mobile device 100 does not utilize anyGPS data in the identification. In such embodiments, markers 230utilized to identify the object 220 are not location-based.

FIG. 3 illustrates a mobile device 100, specifically the display 130 ofthe mobile device 100, wherein the device 100 has executed an objectrecognition application 125 and an AR presentation application 121 topresent within the display 130 indications of recognized objects withinthe live video stream (i.e., surrounding environment 250). The mobiledevice 100 is configured to rely on markers 230 to identify objects 220that are associated with product offers, products with extendedwarranties, new products and the like, and indicate to the user 210 theidentified objects 220 by displaying an indicator 300 on the mobiledevice display 130 in conjunction with display of the live video stream.As illustrated, if an object 220 does not have any markers 230 (or atleast enough markers 230 to yield object identification), the object 220will be displayed without an associated indicator 300.

The object recognition application 125 may use any type of means inorder to identify desired objects 220. For instance, the objectrecognition application 125 may utilize one or more pattern recognitionalgorithms to analyze objects in the environment 250 and compare withmarkers 230 in data storage 171 which may be contained within the mobiledevice 100 (such as within integrated circuit 180) or externally on aseparate system accessible via the connected network. For example, thepattern recognition algorithms may include decision trees, logisticregression, Bayes classifiers, support vector machines, kernelestimation, perceptrons, clustering algorithms, regression algorithms,categorical sequence labeling algorithms, real-valued sequence labelingalgorithms, parsing algorithms, general algorithms for predictingarbitrarily-structured labels such as Bayesian networks and Markovrandom fields, ensemble learning algorithms such as bootstrapaggregating, boosting, ensemble averaging, combinations thereof, and thelike.

Upon identifying an object 220 within the real-time video stream, the ARpresentation application 121 is configured to superimpose an indicator300 on the mobile device display 130. The indicator 300 is generally agraphical representation that highlights or outlines the object 220 andmay be activatable (i.e., include an embedded link), such that the user210 may “select” the indicator 300 and retrieve information related tothe identified object. The information may include any desiredinformation associated with the selected object and may range from basicinformation to greatly detailed information. In some embodiments, theindicator 300 may provide the user 210 with an internet hyperlink tofurther information on the object 220. The information may include, forexample, all types of media, such as text, images, clipart, video clips,movies, or any other type of information desired. In yet otherembodiments, the indicator 300 information related to the identifiedobject may be visualized by the user 210 without “selecting” theindicator 300.

In embodiments in which the indicator 300 provides an interactive tab tothe user 210, the user 210 may select the indicator 300 by anyconventional means, e.g., keystroke, touch, voice command or the like,for interaction with the mobile device 100. For instance, in someembodiments, the user 210 may utilize an input device 140 such as akeyboard to highlight and select the indicator 300 in order to retrievethe information. In a particular embodiment, the mobile device display130 includes a touch screen that the user may employ to select theindicator 300 utilizing the user's finger, a stylus, or the like.

In some embodiments, the indicator 300 is not interactive and simplyprovides information to the user 210 by superimposing the indicator 300onto the display 130. For example, in some instances it may bebeneficial for the AR presentation application 121 to merely identify anobject 220, e.g., just identify the object's name/title, give briefinformation about the object, etc., rather than provide extensive detailthat requires interaction with the indicator 300. The AR presentationapplication 121 is capable of being tailored to a user's desiredpreferences.

Furthermore, the indicator 300 may be displayed at any size on themobile device display 130. The indicator 300 may be small enough that itis positioned on or next to the object 220 being identified such thatthe object 220 remains discernible behind the indicator 300.Additionally, the indicator 300 may be semi-transparent or an outline ofthe object 220, such that the object 220 remains discernible behind orenclosed by the indicator 300. In other embodiments, the indicator 300may be large enough to completely cover the object 220 portrayed on thedisplay 130. Indeed, in some embodiments, the indicator 300 may cover amajority or the entirety of the mobile device display 130.

The user 210 may opt to execute the object recognition application 125and AR presentation application 121 at any desired moment and beginvideo capture and analysis. However, in some embodiments, the objectrecognition application 125 and AR presentation application 121 includesan “always on” feature in which the mobile device 100 is continuouslycapturing video and analyzing the objects 220 within the video stream.In such embodiments, the object recognition application 125 may beconfigured to alert the user 210 that a particular object 220 has beenidentified. The user 210 may set any number of user preferences totailor the object recognition and AR presentation experience to theirneeds. For instance, the user 210 may opt to only be alerted if acertain particular object 220 is identified. Additionally, it will beappreciated that the “always on” feature in which video is continuouslycaptured may consume the mobile device power source 115 more quickly.Thus, in some embodiments, the “always on” feature may disengage if adetermined event occurs such as low power source 115, low levels oflight for an extended period of time (e.g., such as if the mobile device100 is in a user's pocket obstructing a clear view of the environment250 from the mobile device 100), if the mobile device 100 remainsstationary (thus receiving the same video stream) for an extended periodof time, the user sets a certain time of day to disengage, etc.Conversely, if the “always on” feature is disengaged due to theoccurrence of such an event, the user 210 may opt for the “always on”feature to re-engage after the duration of the disengaging event (e.g.,power source 115 is re-charged, light levels are increased, etc.).

In some embodiments, the user 210 may identify objects 220 that theobject recognition application 125 does not identify and add it to thedata storage 171 with desired information in order to be identifiedand/or displayed in the future. For instance, the user 210 may select anunidentified object 220 and enter a name/title and/or any other desiredinformation for the unidentified object 220. In such embodiments, theobject recognition application 125 may detect/record certain markers 230about the object so that the pattern recognition algorithm(s) (or otheridentification means) may detect the object 220 in the future.Furthermore, in cases where the object information is within the datastorage 171, but the object recognition application 125 fails toidentify the object 220 (e.g., one or more identifying characteristicsor markers 230 of the object has changed since it was added to the datastorage 171 or the marker 230 simply was not identified), the user 210may select the object 220 and associate it with an object 220 alreadystored in the data storage 171. In such cases, the object recognitionapplication 125 may be capable of updating the markers 230 for theobject 220 in order to identify the object 220 in future video streams.

In addition, in some embodiments, the user 210 may opt to edit theinformation or add to the information provided by the indicator 300. Forinstance, the user 210 may opt to include user-specific informationabout a certain object 220 such that the information may be displayedupon a future identification of the object 220. Conversely, in someembodiments, the user may opt to delete or hide an object 220 from beingidentified and an indicator 300 associated therewith being displayed onthe mobile device display 130.

Furthermore, in some instances, an object 220 may include one or moremarkers 230 identified by the object recognition application 125 thatleads the object recognition application 125 to associate an object 220with more than one objects 220 in the data storage 171. In suchinstances, the user 210 may be presented with multiple candidateidentifications and may opt to choose the appropriate identification orinput a different identification. The multiple candidates may bepresented to the user 210 by any means. For instance, in one embodiment,the candidates are presented to the user 210 as a list wherein the“strongest” candidate is listed first based on reliability of theidentification. Upon input by the user 210 identifying the object 220,the object recognition application 125 may “learn” from the input andstore additional markers 230 in order to avoid multiple identificationcandidates for the same object 220 in future identifications.

Additionally, the object recognition application 125 may utilize othermetrics for identification than identification algorithms. For instance,the object recognition application 125 may utilize the user's location,time of day, season, weather, speed of location changes (e.g., walkingversus traveling), “busyness” (e.g., how many objects are in motionversus stationary in the video stream), as well any number of otherconceivable factors in determining the identification of objects 220.Moreover, the user 210 may input preferences or other metrics for whichthe object recognition application 125 may utilize to narrow results ofidentified objects 220.

In some embodiments, the AR presentation application 121 may have theability to gather and report user interactions with displayed indicators300. The data elements gathered and reported may include, but are notlimited to, number of offer impressions; time spent “viewing” an offer,product, object or business; number of offers investigated via aselection; number of offers loaded to an electronic wallet and the like.Such user interactions may be reported to any type of entity desired. Inone particular embodiment, the user interactions may be reported to afinancial institution and the information reported may include customerfinancial behavior, purchase power/transaction history, and the like.

In various embodiments, information associated with or related to one ormore objects 220 that is retrieved for presentation to a user 210 viathe mobile device 100 may be permanently or semi-permanently associatedwith the object 220. In other words, the object 220 may be “tagged” withthe information. In some embodiments, a location pointer is associatedwith an object 220 after information is retrieved regarding the object220. In this regard, subsequent mobile devices capturing the object 220for recognition may retrieve the associated information, tags and/orpointers in order to more quickly retrieve information regarding theobject 220. In some embodiments, the mobile device 100 provides the user210 an opportunity to post messages, links to information or the likeand associate such postings with the object 220. Subsequent users maythen be presenting such postings when their mobile devices capture andrecognize an object 220. In some embodiments, the information gatheredthrough the recognition and information retrieval process may be postedby the user 210 in association with the object 220. Such tags and/orpostings may be stored in a predetermined memory and/or database forease of searching and retrieval.

In an exemplary embodiment, the AR presentation application 121 iscapable of analyzing the surrounding environment 250 in search ofproducts with embedded sensors FIG. 4 illustrates a high-level flowdiagram of a method 400 for analyzing the surrounding environment 250 insearch of products with embedded sensors according to an embodiment ofthe invention.

At block 410, a video stream of the surrounding environment 250 isanalyzed in search of at least one product with an embedded sensor. Asused herein, an “embedded sensor” is a device associated with a productin such a way that the sensors may monitor, obtain, collect, emit,and/or indicate information about the product. In one embodiment, theembedded sensors may include, at least in part, an RFID tag, a chip, aradio signal emitter, an invisible marker, etc. In another embodiment,an embedded tag includes a nanosensor. For example, the nanosensors ormicrosensors may be configured to measure acceleration, conductivity,electromagnetic field, friction, light, movement, quantity, radiation,sound, stress, temperature, time, vibration, viscosity, weight, or someother measurable physical quantity or characteristic. The sensors may becapable of wireless transmission of such information, eithercontinuously or intermittently. In one embodiment, the sensors mayautomatically transmit the information, whereas in another embodiment,the sensors may transmit the information in response to beinginterrogated. The information transmitted from an embedded sensor mayinclude, for instance, “defaults” with the product. As used herein, a“default” may be anything concerning the product that may lead to itsreplacement or maintenance. For instance, a default may include a foodproduct that has reached its expiration date (i.e., replacement needed)or a shortage in the circuitry of an electronic-based product (i.e.,maintenance or repair services needed).

The mobile device 100 may be capable of detecting or sensing embeddedsensors within products. The detection may utilize any marker 230 asdescribed above in the identification of the product. However, in oneembodiment, the only marker 230 required for product identification isthe embedded sensors. In such embodiments, generally, the embeddedsensor emits a wireless signal which is detectable to the ARpresentation application 121 of the mobile device 100.

Block 420 illustrates the step of identifying one or more productswithin the video stream that has one or more embedded sensors associatedtherewith. As the video stream is analyzed (block 410), the ARpresentation application 121 may detect an embedded sensors emitting awireless signal (e.g., an RF signal).

At block 430, the mobile device 100 retrieves the information associatedwith the products from the embedded sensors. In one embodiment, the ARpresentation application 121 is capable of receiving information frommultiple embedded sensors at once. For example, a user 210 may utilizethe AR presentation application 121 to scan the pantry or refrigerator,for instance, and receive information from a plurality of products atonce. As noted above, the information may include data concerning theproducts such that the user 210 may determine what action needs to betaken (e.g., product replacement).

At block 440, the AR presentation application 121 may presentinformation to the user via the mobile device display of the videostream. For instance, a indicator 300 may be superimposed onto the videostream on the display 130 of the mobile device 100. The indicator 300may be interactive. In one embodiment, the indicator 300 may include theinformation transmitted from the embedded sensors explicitly (e.g.,amount of product remaining, expiration date, etc.). However, in oneembodiment, the AR presentation application 121 may present informationto the user 210 that is derived from the information received from theembedded sensors. For example, a user 210 that utilizes the ARpresentation application 121 to analyze the products within arefrigerator may opt to have a shopping list automatically generatedbased on preset preferences that determine when certain products need tobe replaced.

In one embodiment, the mobile device 100 is operated in the “always on”mode to continuously analyze the surrounding environment 250 to detectproduct embedded sensors. Thus, in one embodiment, the user 210 activelyutilizes the mobile device 100 in search of products with embeddedsensors by viewing the mobile device display 130 as the user 210 pansaround scanning for products. However, in another embodiment, the user210 passively utilizes the mobile device 100 in search of predeterminedobjects. In such embodiments, the mobile device 100 may simply beaffixed to or otherwise carried by the user 210 and the mobile device100 simply scans the images captured in search of products with embeddedsensors.

The mobile device 100 may alert the user 210 to important or criticalinformation associated with one or more products. As used herein,“important” or “critical” information may be defined by the user 210,but will generally include information concerning product defaults orimpending product defaults. The user 210 may opt to input preferences asto when to be alerted of an impending product default such as thequantity of product is approaching a predetermined threshold or theexpiration date of a product is approaching. Similarly, as noted above,a product default may include something that is irregular with theproduct such as a power source malfunction, a drop in efficiency,hardware/software malfunctions, etc. The “alert” to the user 210 may beone or more indicators that appear on the mobile device display 130.However, when the AR presentation application 121 is operated in apassive mode (i.e., the user 210 is not attentive to and watching thevideo stream being analyzed), it is generally preferable that the alertsbe of audible signals, vibration of the mobile device 100, Short MessageService/text messages, emails or the like which may identify the object,provide access to an offer or direct the user to the video stream(real-time or recorded) for access to the displayed indicators 300.

The “product” detectable to the AR presentation application 121 willgenerally include any product that utilizes an embedded sensors capableof emitting a wireless signal. In one embodiment, the product is a foodproduct. In another embodiment, the product is a household item such asa household appliance. In still further embodiments, the product may bean automobile, an electronic device, a non-food consumer product, etc.

In a particular embodiment, the indicator 300 displayed to the user 210is interactive and may present the user 210 with information regardingpurchasing the product. For instance, the information may include costof replacement, estimated time to failure, nearest retailer stocking theproduct, estimated delivery time, etc. Additionally, the user 210 may bepresented with the option to purchase the product. For instance, theindicator 300 displayed on the mobile device 100 may include a link,such as, for example, an internet hyperlink, that gives the user 210 theoption to purchase the product.

FIG. 5 illustrates a high-level mixed block and flow diagram of a system500 for accessing product information from embedded sensors via a videostream captured by a mobile device 100. It will be understood that thesystem 500 illustrated is an exemplary embodiment. As shown, the system500 includes a user 210, a mobile device 100, and a financialinstitution.

As represented by block 505, the user 210 activates the AR presentationapplication 121 of the mobile device 100. The AR presentationapplication 121 may be operated in active mode or passive mode asdiscussed above. Once the mobile device 100 loads the AR presentationapplication 121, the mobile device 100 initiates scanning of theenvironment 250 to search for the products with embedded sensors, asrepresented by block 515.

At block 520, the mobile device 100 then may identify a product withinthe surrounding environment 250. Upon identifying a product, the mobiledevice 100 alerts the user 210 that the product has been identified asshown in block 525. The alert may be any way to bring the identificationto the user's attention. In a particular embodiment, the mobile devicedisplay 130 may include an indicator and the mobile device 100 may emitan audible sound and/or vibration or other type of alert as discussedabove.

In some embodiments, the information provided by the real-time videostream may be compared to data provided to the system through an API. Inthis way, the data may be stored in a separate application and beimplemented by request from the mobile device and/or server.

At block 530, the mobile device 100 may present information associatedwith the product to the user 210, and as illustrated at block 535, theuser 210 may receive the information associated with the object. Theinformation displayed to the user may range from a simple identificationof the object to a large amount of data associated with the object,depending, at least in part, on the sophistication of the embeddedsensors.

In a particular embodiment, the information displayed to the user 210may include product purchasing information. In such embodiments, theuser may opt to purchase the product as shown in block 540. The user 210may opt to utilize any feasible means in purchasing the product. In someembodiments, the business that sells the product may be capable ofaccepting online orders for the identified object via the Internet, anintranet or other wireless connection utilizing the mobile device 100.In a particular embodiment, the mobile device 100 includes a mobilewallet 800 (FIG. 8) and the business is capable of receiving paymentvehicle data from the user's mobile device 100 via a contactless paymentterminal 700 (FIGS. 6 and 7) as discussed further below. Thus, asillustrated at block 545, the mobile device 100 may transmit paymentdata for purchase of the product.

As represented by block 550, a financial institution may monitor andrecord financial data and/or video image analysis user data associatedwith the object identification. For instance, the data the financialinstitution may record includes the amount of time the user reviewed thepresented information, whether the user 210 was presented with an optionto purchase, whether the user 210 purchased the product, the price ofthe product, the frequency in which the user utilizes the ARpresentation application 121 to identify and purchase objects, etc. Suchinformation may be beneficial to the financial institution indetermining future business strategies.

The mobile device 100 may be equipped to provide the user 210 theability to transmit payment information directly from the mobile device100 to a payment terminal in order to complete a purchase. Referring toFIG. 6, a block diagram illustrating a contactless transactionenvironment 500 configured for making a contactless transaction via amobile device 100 is shown. As illustrated, the contactless transactionenvironment 600 may include a mobile device 100 operable by a user 210who may be a customer who wishes to make a contactless payment for theidentified product. The contactless transaction environment 600 may alsoinclude a contactless terminal 700 that may be automated or may beoperable by a cashier 610. The contactless terminal 700 may permit auser 210 to make a contactless payment with the mobile device 100.

Of course, the environment 600 may also include a contact terminal 620that may permit a user 210 to make a payment via a contact transactiondevice such as a payment card that has a magnetic stripe which may beswiped through the contact terminal 620.

The contactless transaction environment 600 may also include aworkstation 640 and a processing system 650 that are in electroniccommunication with the contactless terminal 700 via a network 630, whichmay be the Internet, an intranet or the like. The user interface 615situated on the contactless terminal 700 may be any sort of device, suchas light emitting diodes (LEDs) for indicating that payment has beenreceived, invalid payment vehicle, exceeded credit limit, etc.

In FIG. 6, the network 630 may include a local area network (LAN), awide area network (WAN), and/or a global area network (GAN). The network630 may provide for wireline, wireless, or a combination of wireline andwireless communication between devices in the network. In someembodiments, the network 630 includes the Internet. In some embodiments,the network 630 may include a wireless telephone network.

FIG. 7 illustrates an embodiment of a contactless terminal 700 that isdepicted in FIG. 6. The contactless terminal 700 may include variousfeatures, such as a network communication interface 710, a processingdevice 730, a transceiver interface 720, and a memory device 740 thatmay include a transceiver application 750.

As used with respect to the contactless terminal 700, a “communicationinterface” may generally include a modem, server, transceiver, and/orother device for communicating with other devices on a network. Thenetwork communication interface 710 may be a communication interfacehaving one or more communication devices configured to communicate withone or more other devices in the contactless transaction environment600, such as the mobile device 100, the workstation 640, the processingsystem 650, other processing systems, data systems, etc.

In one embodiment, the transceiver interface 720 is a separate modulethat may generally include a transceiver, i.e., one or more antennasand/or other electronic circuitry, devices, and software, for receivingelectronic payment vehicle data when the mobile device 100 is held closeto or “tapped” at the contactless terminal 700. In accordance withembodiments of the invention, the term “tap,” “tapped,” “tapping,” etc.may refer to bringing a mobile device close to or within the proximityof a payment terminal so that information can be communicated wirelesslybetween the mobile device and the payment terminal using short rangewireless transmission technology, such near-field communication (NFC)technology, radio-frequency (RF) technology, or the like. Tapping mayinclude physically tapping the mobile device against an appropriateportion of the payment terminal or it may include only waving or holdingthe mobile device near an appropriate portion of the payment terminalwithout making physical contact with the payment terminal. In someembodiments, the transceiver interface 720 is part of the networkcommunication interface 710. Furthermore, the transceiver interface 720may also be used as an interface to send data to the mobile device 100when the mobile device 100 is held close to or tapped at the contactlessterminal 700.

An output device for the transceiver interface 720 may include a displaythat provides instructions regarding the steps for making a contactlesstransaction. In some embodiments where the contactless terminal 700requests authentication, such as a signature, the display may also serveas a touchpad input device to input the user's signature via a stylus.Other output devices may include one or more LEDs or an audio speaker,both which may indicate to the user 210 that data has been successfullyreceived from the mobile device 100. A printer that can print paperreceipts may also be incorporated into the contactless terminal 700.Other embodiments of the contactless terminal 700 may carry other inputand output devices, such as a mouse, keyboard, button, touchpad, touchscreen, microphone, speaker, light, joystick, switch, or the like.

As used with respect to the contactless terminal 700, a “processingdevice,” 730 may generally refer to a device or combination of deviceshaving circuitry used for implementing the communication and/or logicfunctions of a particular system. For example, a processing device 730may include a digital signal processor device, a microprocessor device,and various analog-to-digital converters, digital-to-analog converters,and other support circuits and/or combinations of the foregoing. Controland signal processing functions of the system may be allocated betweenthese processing devices according to their respective capabilities. Theprocessing device may further include functionality to operate one ormore software programs based on computer-executable program codethereof, which may be stored in a memory. As the phrase is used herein,a processing device may be “configured to” perform a certain function ina variety of ways, including, for example, by having one or moregeneral-purpose circuits perform the function by executing particularcomputer-executable program code embodied in computer-readable medium,and/or by having one or more application-specific circuits perform thefunction. The processing device 730 may be configured to use the networkcommunication interface 710 and/or the transceiver interface 720 totransmit and/or receive data and/or commands to and/or from the otherdevices that are visible in the contactless payment environment 600.

As used with respect to the contactless terminal 700, a “memory device”may generally refer to a device or combination of devices that store oneor more forms of computer-readable media for storing data and/orcomputer-executable program code/instructions. For example, in oneembodiment, the memory device may include any computer memory thatprovides an actual or virtual space to temporarily or permanently storedata and/or commands provided to the processing device when it carriesout its functions described herein. In one embodiment, the memory devicestores a transceiver application 750. The transceiver application 750may work in conjunction with the previously described transceiverinterface 720 to receive electronic payment vehicle data when the mobiledevice 100 is held close to or tapped at the contactless terminal 700.In some embodiments, the transceiver application 750 may also beconfigured to send data to the mobile device 100 when the mobile device100 is held close to or tapped at the payment terminal 700, orpotentially provide a power source 115 to the mobile device 100 in theevent the primary power source is depleted.

As illustrated in FIG. 6, a contactless terminal 200 may be connected toa workstation 640 via the network 630. The workstation 640 may be usedby the cashier 610 or other personnel to interact with the contactlessterminal 700. The workstation 640 may include various features, such asa network communication interface, a processing device, a userinterface, and a memory device.

As used with respect to the workstation 640, a “communication interface”may generally include a modem, server, transceiver, and/or other devicefor communicating with other devices on a network. The networkcommunication interface may be a communication interface having one ormore communication devices configured to communicate with one or moreother devices on the network 630, such as the contactless terminal 700and/or contact terminal 620, the processing system 650, other processingsystems, data systems, etc.

As used with respect to the workstation 130, a “processing device” maygenerally refer to a device or combination of devices having circuitryused for implementing the communication and/or logic functions of aparticular system as described above. For example, a processing devicemay include a digital signal processor device, a microprocessor device,and various analog-to-digital converters, digital-to-analog converters,and other support circuits and/or combinations of the foregoing. Controland signal processing functions of the system may be allocated betweenthese processing devices according to their respective capabilities. Theprocessing device may further include functionality to operate one ormore software programs based on computer-executable program codethereof, which may be stored in a memory. As the phrase is used herein,a processing device may be “configured to” perform a certain function ina variety of ways, including, for example, by having one or moregeneral-purpose circuits perform the function by executing particularcomputer-executable program code embodied in computer-readable medium,and/or by having one or more application-specific circuits perform thefunction. The processing device may be configured to use a networkcommunication interface and/or the transceiver interface to transmitand/or receive data and/or commands to and/or from the other devicesthat are visible in the network 630.

As used with respect to the workstation 640, a “user interface” maygenerally include a plurality of interface devices and/or software thatallow a user to input commands and data to direct the processing deviceto execute instructions. For example, the user interface may include agraphical user interface (GUI) or an interface to inputcomputer-executable instructions that direct the processing device tocarry out specific functions. The user interface may employ certaininput and output devices to input data received from the user 210 or thecashier 610 or output data to the user or the cashier 610. These inputand output devices may include a display, mouse, keyboard, button,touchpad, touch screen, microphone, speaker, light, joystick, switch,and/or other customer input/output device for communicating with one ormore customers. As used with respect to the workstation 640, a “memorydevice” may generally refer to a device or combination of devices thatstore one or more forms of computer-readable media for storing dataand/or computer-executable program code/instructions. For example, inone embodiment, the memory device may include any computer memory thatprovides an actual or virtual space to temporarily or permanently storedata and/or commands provided to the processing device when it carriesout its functions described herein.

As indicated above, in some embodiments, the user 210 may opt to utilizethe mobile device 100 to wirelessly transmit payment vehicle data to acontactless terminal 700. Typically, the payment vehicle data is storedwithin the mobile device 100 or accessible externally through thenetwork. This payment vehicle data is typically contained within amobile wallet. In accordance with embodiments of the invention, thephrase “mobile wallet” refers to the hardware and/or software in amobile device that enables the mobile device to be used to makecontactless payments at a payment terminal. In accordance withembodiments of the invention, the term “payment vehicle” may refer to anelectronic payment vehicle, such as an electronic credit or debit card.The payment vehicle may not be a “card” at all and may instead beaccount identifying information stored electronically in a mobiledevice, such as in a cell phone.

Turning to FIG. 8, one embodiment of a mobile wallet 800 is illustrated.Importantly, the mobile wallet 800 of the present invention is notlimited to the embodiments shown in the figures. Indeed, the manner inwhich the “menu” of payment vehicles 810 is presented to the user or thetype of additional information stored within the mobile wallet 800 isnot essential to the present invention.

As illustrated, information contained within the mobile wallet 800 andcapable of being transmitted by the mobile device 100 may include thepayment vehicle type 810, the unique payment vehicle account number 820associated with the payment vehicle 810, the name associated with thepayment vehicle 830, the payment vehicle's expiration date 840, thepayment vehicle's security code 850, whether the payment vehicle iscredit or debit 860 and the default payment vehicle 870 for use intransactions.

Thus, methods, systems, computer programs and the like are hereindisclosed that provide for using video analysis to process video streamsand, more specifically, to access product information from embeddedsensors and present information associated with the product to the user,which information may be presented via AR or the like. The methods,apparatuses, systems, and computer program products may additionallyprovide for alerting the user to important or critical informationassociated with one or more products. Many of the embodiments hereindisclosed leverage financial institution data, which is uniquelyspecific to financial institution, in providing information to mobiledevice users in connection with video stream analysis.

While the foregoing disclosure discusses illustrative embodiments, itshould be noted that various changes and modifications could be madeherein without departing from the scope of the described aspects and/orembodiments as defined by the appended claims. Furthermore, althoughelements of the described aspects and/or embodiments may be described orclaimed in the singular, the plural is contemplated unless limitation tothe singular is explicitly stated. Additionally, all or a portion of anyembodiment may be utilized with all or a portion of any otherembodiment, unless stated otherwise. In this regard, the term“processor” and “processing device” are terms that are intended to beused interchangeably herein and features and functionality assigned to aprocessor or processing device of one embodiment are intended to beapplicable to or utilized with all or a portion of any other embodiment,unless stated otherwise.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other changes,combinations, omissions, modifications and substitutions, in addition tothose set forth in the above paragraphs, are possible. Those skilled inthe art will appreciate that various adaptations and modifications ofthe just described embodiments can be configured without departing fromthe scope and spirit of the invention. Therefore, it is to be understoodthat, within the scope of the appended claims, the invention may bepracticed other than as specifically described herein.

The systems, methods, computer program products, etc. described herein,may be utilized or combined with any other suitable AR-relatedapplication. Non-limiting examples of other suitable AR-relatedapplications include those described in the following U.S. ProvisionalPatent Applications, the entirety of each of which is incorporatedherein by reference:

U.S. Provisional Ser. No. Filed On Title 61/450,213 Mar. 8, 2011Real-Time Video Image Analysis Applications for Commerce Activity61/478,409 Apr. 22, 2011 Presenting Offers on a Mobile CommunicationDevice 61/478,412 Apr. 22, 2011 Real-Time Video Analysis for RewardOffers 61/478,394 Apr. 22, 2011 Real-Time Video Image Analysis forProviding Targeted Offers 61/478,399 Apr. 22, 2011 Real-Time AnalysisInvolving Real Estate Listings 61/478,402 Apr. 22, 2011 Real-Time VideoImage Analysis for an Appropriate Payment Account 61/478,405 Apr. 22,2011 Presenting Investment-Related Information on a Mobile CommunicationDevice 61/478,393 Apr. 22, 2011 Real-Time Image Analysis for MedicalSavings Plans 61/478,397 Apr. 22, 2011 Providing Data Associated WithRelationships Between Individuals and Images 61/478,408 Apr. 22, 2011Identifying Predetermined Objects in a Video Stream Captured by a MobileDevice 61/478,400 Apr. 22, 2011 Real-Time Image Analysis for ProvidingHealth Related Information 61/478,403 Apr. 22, 2011 Providing SocialImpact Information Associated With Identified Products or Businesses61/478,407 Apr. 22, 2011 Providing Information Associated With anIdentified Representation of an Object 61/478,415 Apr. 22, 2011Providing Location Identification of Associated Individuals Based onIdentifying the Individuals in Conjunction With a Live Video Stream61/478,419 Apr. 22, 2011 Vehicle Recognition 61/478,417 Apr. 22, 2011Collective Network of Augmented Reality Users 61/508,985 Jul. 18, 2011Providing Information Regarding Medical Conditions 61/508,946 Jul. 18,2011 Dynamically Identifying Individuals From a Captured Image61/508,980 Jul. 18, 2011 Providing Affinity Program Information61/508,821 Jul. 18, 2011 Providing Information Regarding SportsMovements 61/508,850 Jul. 18, 2011 Assessing EnvironmentalCharacteristics in a Video Stream Captured by a Mobile Device 61/508,966Jul. 18, 2011 Real-Time Video Image Analysis for Providing VirtualLandscaping 61/508,969 Jul. 18, 2011 Real-Time Video Image Analysis forProviding Virtual Interior Design 61/508,971 Jul. 18, 2011 Real-TimeVideo Image Analysis for Providing Deepening Customer Value 61/508,764Jul. 18, 2011 Conducting Financial Transactions Based on Identificationof Individuals in an Augmented Reality Environment 61/508,973 Jul. 18,2011 Real-Time Video Image Analysis for Providing Security 61/508,976Jul. 18, 2011 Providing Retail Shopping Assistance 61/508,944 Jul. 18,2011 Recognizing Financial Document Images

1. A method for accessing product information from embedded sensorscomprising: analyzing, via a processing device, a video streamrepresenting the surrounding environment captured on a mobile device;identifying one or more products within the video stream that has one ormore embedded sensors associated therewith; retrieving informationassociated with the one or more products from the one or more embeddedsensors; and presenting, via a display of the video stream on the mobiledevice, the information retrieved, or information derived from theinformation retrieved, from the one or more embedded sensors.
 2. Themethod of claim 1, wherein the mobile device is a mobile phone.
 3. Themethod of claim 1, wherein the mobile device comprises a camera capableof image and/or video capture.
 4. The method of claim 1, wherein the oneor more embedded sensors emit a radio frequency signal.
 5. The method ofclaim 1, wherein the information presented is displayed as a virtualobject overlaying a real-time video stream of the surroundingenvironment.
 6. The method of claim 5, wherein the virtual object isinteractive.
 7. The method of claim 1, wherein the mobile device isoperated in the “always on” mode to continuously analyze the surroundingenvironment.
 8. The method of claim 1, further comprising alerting theuser to critical information associated with one or more products. 9.The method of claim 8, wherein the critical information comprisesinformation regarding a product default or an impending product default.10. The method of claim 9, wherein the product default is the depletionor expiration of the product requiring replacement.
 11. The method ofclaim 9, wherein the product default is a mechanical default requiringmaintenance.
 12. The method of claim 8, wherein the step of alerting theuser comprises producing an audible sound, vibration, generating a ShortMessage Service or text message, generating an email, or combinationsthereof.
 13. The method of claim 1, wherein the one or more products isa food product.
 14. The method of claim 1, wherein the one or moreproducts is a household item.
 15. The method of claim 14, wherein thehousehold item is a household appliance.
 16. The method of claim 1,further comprising presenting the user with an option to purchase theone or more products via the mobile device.
 17. The method of claim 1,further comprising transmitting payment information from the mobiledevice to a seller of the one or more products in order to purchase theproduct.
 18. The method of claim 1, further comprising transmittingpayment information via a wireless connection between the mobile deviceand a contactless payment terminal to purchase a product or service. 19.The method of claim 1, further comprising transmitting financial dataand/or other user-specific data to a financial institution.
 20. A systemfor accessing product information from embedded sensors comprising: aprocessing device configured for: analyzing a video stream representingthe surrounding environment captured on a mobile device; identifying oneor more products within the video stream that has one or more embeddedsensors associated therewith; retrieving information associated with theone or more products from the one or more embedded sensors; andpresenting, via a display of the video stream on the mobile device, theinformation retrieved, or information derived from the informationretrieved, from the one or more embedded sensors.
 21. The system ofclaim 20, wherein the mobile device is a mobile phone.
 22. The system ofclaim 20, wherein the mobile device comprises a camera capable of imageand/or video capture.
 23. The system of claim 20, wherein the one ormore embedded sensors emit a radio frequency signal.
 24. The system ofclaim 20, wherein the information presented is displayed as a virtualobject overlaying a real-time video stream of the surroundingenvironment.
 25. The system of claim 24, wherein the virtual object isinteractive.
 26. The system of claim 20, wherein the mobile device isoperated in the “always on” mode to continuously analyze the surroundingenvironment.
 27. The system of claim 20, wherein the processing deviceis further configured to alert the user to critical informationassociated with one or more products.
 28. The system of claim 27,wherein the critical information comprises information regarding aproduct default or an impending product default.
 29. The system of claim28, wherein the product default is the depletion or expiration of theproduct requiring replacement.
 30. The system of claim 28, wherein theproduct default is a mechanical default requiring maintenance.
 31. Thesystem of claim 27, wherein alerting the user comprises producing anaudible sound, vibration, generating a Short Message Service or textmessage, generating an email, or combinations thereof.
 32. The system ofclaim 20, wherein the one or more products is a food product.
 33. Thesystem of claim 20, wherein the one or more products is a householditem.
 34. The system of claim 33, wherein the household item is ahousehold appliance.
 35. The system of claim 20, wherein the processingdevice is further configured to present the user with an option topurchase the one or more products via the mobile device.
 36. The systemof claim 20, wherein the processing device is further configured totransmit payment information from the mobile device to a seller of theone or more products in order to purchase the product.
 37. The system ofclaim 20, wherein the processing device is further configured totransmit payment information via a wireless connection between themobile device and a contactless payment terminal to purchase a productor service.
 38. The system of claim 20, wherein the processing device isfurther configured to transmit financial data and/or other user-specificdata to a financial institution.
 39. A computer program product foraccessing product information from embedded sensors comprising anon-transitory computer readable medium comprising one or morecomputer-readable code portions configured for: analyzing a video streamrepresenting the surrounding environment captured on a mobile device;identifying one or more products within the video stream that has one ormore embedded sensors associated therewith; retrieving informationassociated with the one or more products from the one or more embeddedsensors; and presenting, via a display of the video stream on the mobiledevice, the information retrieved, or information derived from theinformation retrieved, from the one or more embedded sensors.
 40. Thecomputer program product of claim 39, wherein the mobile device is amobile phone.
 41. The computer program product of claim 39, wherein themobile device comprises a camera capable of image and/or video capture.42. The computer program product of claim 39, wherein the one or moreembedded sensors emit a radio frequency signal.
 43. The computer programproduct of claim 39, wherein the information presented is displayed as avirtual object overlaying a real-time video stream of the surroundingenvironment.
 44. The computer program product of claim 43, wherein thevirtual object is interactive.
 45. The computer program product of claim39, wherein the mobile device is operated in the “always on” mode tocontinuously analyze the surrounding environment.
 46. The computerprogram product of claim 39, wherein the one or more computer-readablecode portions is further configured to alert the user to criticalinformation associated with one or more products.
 47. The computerprogram product of claim 46, wherein the critical information comprisesinformation regarding a product default or an impending product default.48. The computer program product of claim 47, wherein the productdefault is the depletion or expiration of the product requiringreplacement.
 49. The computer program product of claim 47, wherein theproduct default is a mechanical default requiring maintenance.
 50. Thecomputer program product of claim 46, wherein alerting the usercomprises producing an audible sound, vibration, generating a ShortMessage Service or text message, generating an email, or combinationsthereof.
 51. The computer program product of claim 39, wherein the oneor more products is a food product.
 52. The computer program product ofclaim 39, wherein the one or more products is a household item.
 53. Thecomputer program product of claim 52, wherein the household item is ahousehold appliance.
 54. The computer program product of claim 39,wherein the one or more computer-readable code portions is furtherconfigured to present the user with an option to purchase the one ormore products via the mobile device.
 55. The computer program product ofclaim 39, wherein the one or more computer-readable code portions isfurther configured to transmit payment information from the mobiledevice to a seller of the one or more products in order to purchase theproduct.
 56. The computer program product of claim 39, wherein the oneor more computer-readable code portions is further configured totransmit payment information via a wireless connection between themobile device and a contactless payment terminal to purchase a productor service.
 57. The computer program product of claim 39, wherein theone or more computer-readable code portions is further configured totransmit financial data and/or other user-specific data to a financialinstitution.